While operating a multi-camera system, there are several scenarios that may cause data loss, these include the following: misbehaving hardware, system performance (host cannot accept all incoming data), and transient/startup conditions especially during camera image sensor enable changes. These scenarios cause the data from one or more stream sources to be lost or be delivered at different instances of time.
Currently, a mechanism for synchronized capture from multiple heterogeneous cameras, array of microphones and other sensors (e.g., positional sensors, environmental sensors, etc.) do not exist to microsecond level accuracy. Therefore, system designers that desire such synchronization must design and build a system using programmable logic, firmware and/or other methods to implement precision synchronization capabilities.
Certain sensor manufacturers provide synchronization mechanisms for sensors of the same type, but do not provide a complete solution for streaming synchronized video data to an application. For example, image sensors such as Aptina's MT9M114 allow synchronized capture from two or more sensors of the same type by connecting them in a daisy chain manner and triggering capture using one of the sensors as the master. However, this mechanism is limited to the same type of sensor and is typically used in stereoscopic image capture applications. It is also limited to aligned captures, with all sensors imaging simultaneously. It does not allow for deliberately skewed captures.
Other sensors expose a general purpose input/output (GPIO) pin or a Frame-Sync pin that can be used to trigger capture on the sensor. But, it is the responsibility of the developer to build a system using this pin to synchronize multiple cameras and aggregate the video streams from each of them.
Today, there are a number of academic papers that describe camera array systems built using specific sensors and hardware and describe high performance imaging using large camera arrays. However, the system design is described for the specific image sensors and their architecture and does not account for modern data streaming interfaces used in mobile devices such as, for example, Mobile Industry Processor Interface MIPI)/Universal Serial Bus 3.0 (USB3.0), etc. The systems they describe use homogenous image sensors and did not address the case of synchronizing to heterogeneous image sensors or additional sensors such as microphone arrays and positional sensors. Furthermore, they do not address generating timestamps in the presence of intermediate image-signal processing units in the data path.